iCrawl: Improving the Freshness of Web Collections by Integrating Social Web and Focused Web Crawling

摘要

Researchers in the Digital Humanities and journalists need to monitor,collect and analyze fresh online content regarding current events such as theEbola outbreak or the Ukraine crisis on demand. However, existing focusedcrawling approaches only consider topical aspects while ignoring temporalaspects and therefore cannot achieve thematically coherent and fresh Webcollections. Especially Social Media provide a rich source of fresh content,which is not used by state-of-the-art focused crawlers. In this paper weaddress the issues of enabling the collection of fresh and relevant Web andSocial Web content for a topic of interest through seamless integration of Weband Social Media in a novel integrated focused crawler. The crawler collectsWeb and Social Media content in a single system and exploits the stream offresh Social Media content for guiding the crawler.